config: add rust build configurations #1468

10ne1 · 2022-10-12T14:38:44Z

config: add rust build configurations

Starting with the upcoming v6.1, mainline Linux has merged the
initial Rust infrastructure so this adds some configs for build
testing it together with some sample modules.

The kernel requires a specific version of rustc, so we add the
rustc-1.62 build environment which derives from clang-15, since
a C compiler is still required to build the kernel and the
supported kernel version is 15 (we might bump this later).

Obviously GCC can be used as well but for now testing all the
toolchain combinations does not add significant value. In the
future more toolchain combinations can be used as needed.

The official "offline" toolchain installation method is used as
documented at [1] with sha256sum and because some distros like
Debian stable might not provide up to date toolchain and crates
to keep up with the mainline kernel.

Only the x86_64 architecture is supported by the kernel for now.

We also add the Rust-for-Linux kernel maintainer trees which
contain additional modules and bindings.

Signed-off-by: Adrian Ratiu <[email protected]>

config/core/build-configs.yaml

nuclearcat · 2022-10-17T08:22:25Z

Testing it, might take a while, i will report progress here.
Docker image rustc-1.62:kernelci built successfully

mgalka

In general it looks good. I'd probably go with 2 separate names for the .config files (mentioned above)

config/core/build-configs.yaml

mgalka

Some changes are required to make the image build by kernelci Jenkins.

config/core/build-configs.yaml

Suggested by Michal in pull request kernelci#1468 Co-authored-by: Michał Gałka <[email protected]>

Suggested by Michal in pull request kernelci#1468 Co-authored-by: Michał Gałka <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

gctucker · 2022-11-07T19:16:15Z

See also the email thread with @ojeda from January 2022 https://groups.io/g/kernelci/message/1336

ojeda · 2022-11-07T19:25:00Z

Thanks for the Cc!

The official "rustup" toolchain installation method is used as
documented in the upstream kernel

The current commits seem to be using the standalone installers, not rustup. Is the commit description correct?

mgalka · 2022-11-07T19:43:13Z

Tested builds on staging:
https://bot.staging.kernelci.org/job/kernel-build/91146/console
https://bot.staging.kernelci.org/job/kernel-build/91160/consoleFull

Builds OK

config/docker/rustc-1.62.jinja2

10ne1 · 2022-11-11T11:38:17Z

Hi @ojeda If you are ok with the current state of the PR can you please leave a comment so I can add a reviewed-by tag for you? (I also updated the commit msg & PR description to mention the offline install method).

nuclearcat · 2022-11-11T15:10:38Z

I'm seeing some problems with rust on staging:

+ kci_build pull_tarball --url http://storage.staging.kernelci.org/rust-for-linux/rust/v6.1-rc1-1486-g729c03011c60/linux-src_rust-for-linux.tar.gz --retries 3 --delete
+ kci_build generate_fragments --build-config=rust-for-linux
kernel/configs/rust.config
kernel/configs/rust-for-linux-samples.config
kernel/configs/amdgpu.config
kernel/configs/crypto.config
kernel/configs/ima.config
kernel/configs/kvm_guest.config
kernel/configs/x86-chromebook.config
kernel/configs/xfstests.config
+ kci_build init_bmeta --build-config=rust-for-linux --commit=729c03011c60382030520f84904a85e8a5b3cfb9 --describe=v6.1-rc1-1486-g729c03011c60 --describe-verbose=v6.1-rc1-1486-g729c03011c60
Initialising build meta-data in /tmp/kci/linux/build
Install directory: /tmp/kci/linux/build/_install_
+ kci_build make_config --defconfig=x86_64_defconfig+x86-chromebook+kselftest
Traceback (most recent call last):
  File "/usr/local/bin/kci_build", line 4, in <module>
    __import__('pkg_resources').run_script('kernelci==1.1', 'kci_build')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 651, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 451, in <module>
    status = opts.command(configs, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 344, in __call__
    res = step.run(args.j, args.verbose, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1204, in run
    self._gen_kci_frag(configs, fragments, kci_frag_name)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1118, in _gen_kci_frag
    with open(path) as frag:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/kci/linux/kernel/configs/kselftest.config'
Build result: 1

bmeta.json

Several rust builds failed.

10ne1 · 2022-11-14T11:30:16Z

@nuclearcat thanks for the heads-up, I've forgot to add the new ${CARGO_HOME}/bin to the path, so bindgen couldn't be found. It's fixed now.

config/docker/rustc-1.62.jinja2

ojeda · 2022-11-14T19:16:52Z

can you please leave a comment so I can add a reviewed-by tag for you?

I wrote a nit above and also I left a comment a few days ago above on minimizing the RUN layers, but that is just an optimization, and if you are not doing it in other files, I would ignore it here (I saw it in qemu.ninja2, but not in others, so I don't know if you are following that practice in KernelCI or not...).

Other than those, it looks good to me!

Reviewed-by: Miguel Ojeda [email protected]

One final note: the rust branch will likely go away or be repurposed at some point in the future -- I assume this is not a problem, but just so that you are aware since you were testing it here :)

Thanks a lot for this!

config/core/build-configs.yaml

nuclearcat · 2022-11-16T12:25:37Z

It seems build still failed recently on staging due kfselftest fragment missing (not sure why):

+ kci_build init_bmeta --build-config=rust-for-linux --commit=729c03011c60382030520f84904a85e8a5b3cfb9 --describe=v6.1-rc1-1486-g729c03011c60 --describe-verbose=v6.1-rc1-1486-g729c03011c60
Initialising build meta-data in /tmp/kci/linux/build
Install directory: /tmp/kci/linux/build/_install_
+ kci_build make_config --defconfig=x86_64_defconfig+x86-chromebook+kselftest
Traceback (most recent call last):
  File "/usr/local/bin/kci_build", line 4, in <module>
    __import__('pkg_resources').run_script('kernelci==1.1', 'kci_build')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 651, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 451, in <module>
    status = opts.command(configs, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 344, in __call__
    res = step.run(args.j, args.verbose, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1204, in run
    self._gen_kci_frag(configs, fragments, kci_frag_name)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 1118, in _gen_kci_frag
    with open(path) as frag:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/kci/linux/kernel/configs/kselftest.config'
+ export KERNEL_BUILD_RESULT=1
+ KERNEL_BUILD_RESULT=1
+ set +x
Build result: 1

bmeta.json
{
    "build": {
        "duration": 0.03600597381591797,
        "status": "PASS"
    },
    "environment": {
        "arch": "x86_64",
        "compiler": "clang",
        "compiler_version": "15",
        "compiler_version_full": "Debian clang version 15.0.4-++20221102053030+5c68a1cb1231-1~exp1~20221102053120.92",
        "cross_compile": "",
        "cross_compile_compat": "",
        "make_opts": {
            "KBUILD_BUILD_USER": "KernelCI"
        },
        "name": "rustc-1.62",
        "platform": {
            "uname": [
                "Linux",
                "build-j92564-x86-64-rustc-1-62-x86-64-defconfig-x86-chromed54hv",
                "5.10.127+",
                "#1 SMP Fri Jul 8 00:39:24 UTC 2022",
                "x86_64",
                ""
            ]
        },
        "use_ccache": true
    },
    "revision": {
        "branch": "rust",
        "commit": "729c03011c60382030520f84904a85e8a5b3cfb9",
        "describe": "v6.1-rc1-1486-g729c03011c60",
        "describe_verbose": "v6.1-rc1-1486-g729c03011c60",
        "tree": "rust-for-linux",
        "url": "https://github.com/Rust-for-Linux/linux.git"
    }
}
_install_

10ne1 · 2022-11-17T12:37:10Z

@nuclearcat This fails because the rust configurations do not depend on kselftest config fragment generation, but apparently kselftest is used by the staging tests? Why do they do that?

We could fix it the following way, but I don't think it makes sense because kselftests should not be required for this PR.

diff --git a/config/core/build-configs.yaml b/config/core/build-configs.yaml
index 98809ab7..8cf16c8c 100644
--- a/config/core/build-configs.yaml
+++ b/config/core/build-configs.yaml
@@ -1023,7 +1023,7 @@ build_configs:
     variants:
       rustc-1.62:
         build_environment: rustc-1.62
-        fragments: [rust, rust-for-linux-samples]
+        fragments: [rust, rust-for-linux-samples, kselftest]
         architectures:
           x86_64: *x86_64_arch

Then the following works:

./kci_build init_bmeta --build-config=rust-for-linux
./kci_build generate_fragments --build-config=rust-for-linux
./kci_build make_config --defconfig=x86_64_defconfig+x86-chromebook+kselftest

But again, I repeat, kselftest should not be required. Is there any way to avoid it? To drop that hardcoded +kselftest from the kci_build make_config invocation.

10ne1 · 2022-11-23T11:54:51Z

But again, I repeat, kselftest should not be required. Is there any way to avoid it? To drop that hardcoded +kselftest from the kci_build make_config invocation.

@nuclearcat @mgalka Gentle ping. Any thoughts about dropping the unnecessary +kselftest from the staging run kci_build make_config invocation?

nuclearcat · 2022-11-26T16:35:41Z

I am not sure why it is hardcoded, @mgalka @gctucker ?

mgalka · 2022-11-28T11:45:59Z

I am not sure why it is hardcoded, @mgalka @gctucker ?

I don't know why +keselftest is part of the x86_64_arch. I think we can just add config without kselftest for rust builds.

  rust:
    tree: mainline
    branch: 'master'
    variants:
      rustc-1.62:
        build_environment: rustc-1.62
        fragments: [rust, rust-samples]
        architectures:
          x86_64: &x86_64_arch_nokselftest
            base_defconfig: 'x86_64_defconfig'
            extra_configs:
              - 'allmodconfig'
              - 'allnoconfig'
              - 'x86_64_defconfig+x86-chromebook'
              - 'x86_64_defconfig+x86-chromebook+amdgpu'
            fragments: [amdgpu, crypto, ima, x86_kvm_guest, x86-chromebook]

  rust-for-linux:
    tree: rust-for-linux
    branch: 'rust'
    variants:
      rustc-1.62:
        build_environment: rustc-1.62
        fragments: [rust, rust-for-linux-samples]
        architectures:
          x86_64: *x86_64_arch_nokselftest

I'm giving a try to at the moment on staging.

gctucker · 2022-11-28T11:52:23Z

Well staging build configurations are pretty arbitrary and aim at getting the highest tests / build ratio. Building with kselftest fragment enabled is required to run kselftests, and we need that on staging. There's no point building without kselftest on staging as that's not adding any real coverage. If we can't build kselftest with rust then we just need to fix that in the rust build config options, no need to disable kselftest builds when rust is not enabled.

Starting with the upcoming v6.1, mainline Linux has merged the initial Rust infrastructure so this adds some configs for build testing it together with some sample modules. The kernel requires a specific version of rustc, so we add the rustc-1.62 build environment which derives from clang-15, since a C compiler is still required to build the kernel and the supported kernel version is 15 (we might bump this later). Obviously GCC can be used as well but for now testing all the toolchain combinations does not add significant value. In the future more toolchain combinations can be used as needed. The official "offline" toolchain installation method is used as documented at [1] with sha256sum and because some distros like Debian stable might not provide up to date toolchain and crates to keep up with the mainline kernel. Only the x86_64 architecture is supported by the kernel for now. We also add the Rust-for-Linux kernel maintainer trees which contain additional modules and bindings. [1] https://forge.rust-lang.org/infra/other-installation-methods.html Reviewed-by: Miguel Ojeda <[email protected]> Co-authored-by: Michał Gałka <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

10ne1 · 2022-11-29T08:52:48Z

Well staging build configurations are pretty arbitrary and aim at getting the highest tests / build ratio. Building with kselftest fragment enabled is required to run kselftests, and we need that on staging. There's no point building without kselftest on staging as that's not adding any real coverage. If we can't build kselftest with rust then we just need to fix that in the rust build config options, no need to disable kselftest builds when rust is not enabled.

Thanks for the context, that is what I was missing, so we do have to hardcode something either way to pass the staging tests.

Between the choice of the two hardcoding options - the rust build config options (eg x86_64_arch_nokselftest above) and the kselftest fragment, I'd like us to go with just adding the kselftest fragment.

@mgalka do you agree?

EDIT: I've pushed the change adding the keselftest fragment.

mgalka · 2022-11-29T11:46:08Z

@mgalka do you agree?

I'm fine with it. Giving it a try on staging.

mgalka · 2022-12-01T15:14:17Z

Tested OK on staging. A build which previously failed, works fine now.

https://bot.staging.kernelci.org/job/kernel-build/95009/console

mgalka

LGTM

aliceinwire · 2022-12-02T05:51:29Z

Thanks LGTM

PR kernelci#1468 added the mainline rust build in a separate `rust` build configuration instead of adding to the already existing mainline config which can cause unnecessary builds. Move it in the proper place. Suggested-by: Guillaume Tucker <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

PR kernelci#1468 added the mainline rust build in a separate `rust` build configuration instead of adding to the already existing mainline config which can cause unnecessary builds. Move it in the proper place. Suggested-by: Guillaume Tucker <[email protected]> Reviewed-by: Reviewed-by: Miguel Ojeda <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

PR kernelci#1468 added the mainline rust build in a separate `rust` build configuration instead of adding to the already existing mainline config which can cause unnecessary builds. Move it in the proper place. Fixes: 43875d0 ("config: add rust build configurations") Suggested-by: Guillaume Tucker <[email protected]> Reviewed-by: Miguel Ojeda <[email protected]> Reviewed-by: Alice Ferrazzi <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

PR kernelci#1468 added the mainline rust build in a separate `rust` build configuration instead of adding to the already existing mainline config which can cause unnecessary builds. Move it in the proper place. While at it also change the x86_64_arch anchor to build just the base x86_64_defconfig to avoid unnecessary allmodconfig rebuilds. Fixes: 43875d0 ("config: add rust build configurations") Suggested-by: Guillaume Tucker <[email protected]> Reviewed-by: Miguel Ojeda <[email protected]> Reviewed-by: Alice Ferrazzi <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

PR #1468 added the mainline rust build in a separate `rust` build configuration instead of adding to the already existing mainline config which can cause unnecessary builds. Move it in the proper place. While at it also change the x86_64_arch anchor to build just the base x86_64_defconfig to avoid unnecessary allmodconfig rebuilds. Fixes: 43875d0 ("config: add rust build configurations") Suggested-by: Guillaume Tucker <[email protected]> Reviewed-by: Miguel Ojeda <[email protected]> Reviewed-by: Alice Ferrazzi <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

10ne1 requested review from mgalka and nuclearcat October 12, 2022 14:38

10ne1 self-assigned this Oct 12, 2022

10ne1 commented Oct 12, 2022

View reviewed changes

config/core/build-configs.yaml Outdated Show resolved Hide resolved

10ne1 force-pushed the dev/aratiu/add-rust-trees branch 2 times, most recently from 00b7dce to a7147e0 Compare October 13, 2022 15:19

mgalka reviewed Oct 25, 2022

View reviewed changes

config/core/build-configs.yaml Outdated Show resolved Hide resolved

10ne1 force-pushed the dev/aratiu/add-rust-trees branch from a7147e0 to 22f6c42 Compare October 31, 2022 12:50

10ne1 requested a review from mgalka October 31, 2022 12:51

mgalka suggested changes Nov 3, 2022

View reviewed changes

config/core/build-configs.yaml Show resolved Hide resolved

10ne1 added a commit to 10ne1/kernelci-core that referenced this pull request Nov 4, 2022

config: explicitely set empty arch name

bee6a19

Suggested by Michal in pull request kernelci#1468 Co-authored-by: Michał Gałka <[email protected]>

10ne1 requested a review from mgalka November 4, 2022 11:04

10ne1 added a commit to 10ne1/kernelci-core that referenced this pull request Nov 4, 2022

config: explicitely set empty arch name

02f2b0b

Suggested by Michal in pull request kernelci#1468 Co-authored-by: Michał Gałka <[email protected]> Signed-off-by: Adrian Ratiu <[email protected]>

10ne1 force-pushed the dev/aratiu/add-rust-trees branch from bee6a19 to 02f2b0b Compare November 4, 2022 11:06

ojeda reviewed Nov 8, 2022

View reviewed changes

config/docker/rustc-1.62.jinja2 Outdated Show resolved Hide resolved

config/docker/rustc-1.62.jinja2 Outdated Show resolved Hide resolved

config/docker/rustc-1.62.jinja2 Outdated Show resolved Hide resolved

config/docker/rustc-1.62.jinja2 Outdated Show resolved Hide resolved

10ne1 force-pushed the dev/aratiu/add-rust-trees branch from 02f2b0b to f4f2c20 Compare November 8, 2022 17:17

10ne1 requested a review from ojeda November 8, 2022 17:19

10ne1 force-pushed the dev/aratiu/add-rust-trees branch from f4f2c20 to bc83a3e Compare November 11, 2022 08:55

10ne1 force-pushed the dev/aratiu/add-rust-trees branch from bc83a3e to 40dc7d2 Compare November 14, 2022 11:29

ojeda reviewed Nov 14, 2022

View reviewed changes

config/docker/rustc-1.62.jinja2 Outdated Show resolved Hide resolved

10ne1 force-pushed the dev/aratiu/add-rust-trees branch from 40dc7d2 to 8a6323d Compare November 15, 2022 11:13

ojeda mentioned this pull request Nov 16, 2022

Rust support #814

Closed

ojeda mentioned this pull request Nov 16, 2022

KernelCI wanted features & bugfixes Rust-for-Linux/linux#927

Open

11 tasks

ojeda reviewed Nov 16, 2022

View reviewed changes

config/core/build-configs.yaml Show resolved Hide resolved

10ne1 force-pushed the dev/aratiu/add-rust-trees branch from 8a6323d to 1bf07b9 Compare November 29, 2022 08:52

mgalka approved these changes Dec 1, 2022

View reviewed changes

aliceinwire approved these changes Dec 2, 2022

View reviewed changes

aliceinwire merged commit 43875d0 into kernelci:main Dec 2, 2022

10ne1 deleted the dev/aratiu/add-rust-trees branch December 6, 2022 18:20

gctucker linked an issue Feb 8, 2023 that may be closed by this pull request

Rust support #814

Closed

config: add rust build configurations #1468

config: add rust build configurations #1468

Uh oh!

Conversation

10ne1 commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nuclearcat commented Oct 17, 2022

Uh oh!

mgalka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mgalka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gctucker commented Nov 7, 2022

Uh oh!

ojeda commented Nov 7, 2022

Uh oh!

mgalka commented Nov 7, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

10ne1 commented Nov 11, 2022

Uh oh!

nuclearcat commented Nov 11, 2022

Uh oh!

10ne1 commented Nov 14, 2022

Uh oh!

Uh oh!

ojeda commented Nov 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nuclearcat commented Nov 16, 2022

Uh oh!

10ne1 commented Nov 17, 2022

Uh oh!

10ne1 commented Nov 23, 2022

Uh oh!

nuclearcat commented Nov 26, 2022

Uh oh!

mgalka commented Nov 28, 2022

Uh oh!

gctucker commented Nov 28, 2022

Uh oh!

10ne1 commented Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgalka commented Nov 29, 2022

Uh oh!

mgalka commented Dec 1, 2022

Uh oh!

mgalka left a comment

Choose a reason for hiding this comment

Uh oh!

aliceinwire commented Dec 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

10ne1 commented Oct 12, 2022 •

edited

Loading

ojeda commented Nov 14, 2022 •

edited

Loading

10ne1 commented Nov 29, 2022 •

edited

Loading